Video camera which adopts a focal-plane electronic shutter system

ABSTRACT

A video camera includes an image sensor which repeatedly outputs an object scene image produced on an imaging surface by an exposure operation of a focal-plane electronic shutter system in a raster scanning manner. A post-processing circuit extracts a partial object scene image belonging to an extraction area, out of the object scene image outputted from the image sensor. A moving image based on the extracted partial object scene image is displayed on an LCD monitor by an LCD driver. A motion detecting circuit detects motion of the imaging surface in a direction orthogonal to an optical axis. A position of the extraction area is changed by a CPU so that the motion detected by the motion detecting circuit is compensated. The CPU also changes a shape of the extraction area so that a focal plane distortion is restrained, based on the motion detected by the motion detecting circuit.

CROSS REFERENCE OF RELATED APPLICATION

The disclosures of Japanese Patent Application No. 2008-42281, which wasfiled on Feb. 23, 2008, and No. 2009-8455, which was filed on Jan. 19,2009 are incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a video camera. More specifically, thepresent invention relates to a video camera that performs afocal-plane-distortion correcting process on image data outputted froman image sensor adopting a focal-plane electronic shutter system.

2. Description of the Related Art

According to this type of a video camera (1), an image signal outputtedfrom a CMOS image sensor is applied to a focal-plane-distortioncorrecting circuit. Upon noticing successive three frames, thefocal-plane-distortion correcting circuit performs a linearinterpolation process on the two frames different depending on a pixelposition. Thereby, one frame of image signal on which a focal planedistortion is corrected is produced. Furthermore, according to anothervideo camera (2), a scaling parameter corresponding to an electroniczoom magnification instructed by a zoom key is set to a RAW-data scalingcircuit. The RAW-data scaling circuit performs a scaling processaccording to the scaling parameter on the image data outputted from anA/D converter. On an image display portion, a moving image based on RAWdata outputted from the scaling processing circuit is displayed.However, in the video camera (1), a special circuit as described aboveis required in order to correct a focal plane distortion. Furthermore,in the video camera (2), when an angle of view of the image expressed bythe RAW data on which the scaling process is performed is coincidentwith that of the image displayed on the image display portion, it is notpossible to execute an image-quality correcting process affecting theangle of view, and thus, it is probable that the reproducing performanceof the object scene image deteriorates.

SUMMARY OF THE INVENTION

A video camera according to the present invention, comprises: an imagerfor repeatedly outputting an object scene image produced on an imagingsurface by an exposure operation of a focal-plane electronic shuttersystem in a raster scanning manner; an extractor for extracting apartial object scene image belonging to a designated area, out of theobject scene image outputted from the imager; an outputter foroutputting a moving image based on the partial object scene imageextracted by the extractor; a detector for detecting motion of theimaging surface in a direction orthogonal to an optical axis; a positionchanger for changing a position of the designated area so that themotion detected by the detector is compensated; and a shape changer forchanging a shape of the designated area so that a focal plane distortionis restrained based on the motion detected by the detector.

Preferably, the designated area is a rectangular area having a left sideand a right side, and the shape changer includes an inclination amountchanger for changing inclination amounts of the right side and the leftside based on a horizontal component of the motion detected by thedetector.

More preferably, the detector includes an allocator for allocating aplurality of blocks forming a line in a vertical direction to the objectscene image outputted from the imager, and a motion vector detector forindividually detecting motion vectors of a plurality of block imagesbelonging to the plurality of block allocated by the allocator, and theinclination amount changer includes a function creator for creating afunction for defining the inclination amounts based on horizontalcomponents of the plurality of motion vectors created by the motionvector detector.

In an aspect of the present invention, the number of blocks allocated bythe allocator is determined based on an imaging cycle of the imager anda vibration frequency of the imaging surface.

Preferably, the shape changer includes a first size changer for changinga vertical size of the designated area based on the vertical componentof the motion detected by the detector, and the video camera furthercomprises a second size changer for changing a vertical size of thepartial object scene image extracted by the extractor in associationwith a changing process of the first size changer.

More preferably, a change magnification of the second size changer isequivalent to an inverse number of a change magnification of the firstsize changer.

According to the present invention, an imaging control program productexecuted by a processor of a video camera provided with: an imager forrepeatedly outputting an object scene image produced on an imagingsurface by an exposure operation of a focal-plane electronic shuttersystem in a raster scanning manner; an extractor for extracting apartial object scene image belonging to a designated area, out of theobject scene image outputted from the imager; an outputter foroutputting a moving image based on the partial object scene imageextracted by the extractor; and a detector for detecting motion of theimaging surface in a direction orthogonal to an optical axis, theimaging control program product, comprising: a position changing step ofchanging a position of the designated area so that the motion detectedby the detector is compensated; and a shape changing step of changing ashape of the designated area so that a focal plane distortion isrestrained based on the motion detected by the detector.

According to the present invention, an imaging control method executedby a video camera provided with: an imager for repeatedly outputting anobject scene image produced on an imaging surface by an exposureoperation of a focal-plane electronic shutter system in a rasterscanning manner; an extractor for extracting a partial object sceneimage belonging to a designated area, out of the object scene imageoutputted from the imager; an outputter for outputting a moving imagebased on the partial object scene image extracted by the extractor; anda detector for detecting motion of the imaging surface in a directionorthogonal to an optical axis, the imaging control method, comprising: aposition changing step of changing a position of the designated area sothat the motion detected by the detector is compensated; and a shapechanging step of changing a shape of the designated area so that a focalplane distortion is restrained based on the motion detected by thedetector.

A video camera according to the present invention, comprises: an imagerfor repeatedly outputting an image representing an object scene; areducer for reducing the image outputted from the imager; an extractorfor extracting a partial reduced image belonging to an extraction areahaving a predetermined size, out of the reduced image created by thereducer; and a size changer for changing a size of the reduced imagecreated by the reducer in a range exceeding the predetermined size uponreceipt of a zoom operation.

Preferably, the video camera further comprises: a zoom lens arrangedforwardly of the imager; and a magnification changer for changing amagnification of the zoom lens in the same direction as a changedirection of the size changer in association with the changing processof the size changer.

Preferably, the video camera further comprises: a detector for detectingmotion of an imaging surface in a direction orthogonal to an opticalaxis; and a position changer for changing a position of the extractionarea so that the motion detected by the detector is compensated.

More preferably, the imager includes an exposer for exposing the imagingsurface by a focal-plane electronic shutter system, and the video camerafurther comprises a shape changer for changing a shape of the extractionarea so that a focal plane distortion is restrained based on the motiondetected by the detector.

Further preferably, the extraction area is a rectangular area having aleft side and a right side, and the shape changer changes inclinationamounts of the right side and the left side based on a horizontalcomponent of the motion detected by the detector.

In an aspect of the present invention, the video camera furthercomprises a limiter for limiting a change amount of the position changerby referencing the inclination amounts changed by the shape changer.

More preferably, the video camera further comprises a stopper forstopping the position changer when the motion detected by the detectoris corresponding to a pan/tilt operation of the imaging surface.

Preferably, the image outputted from the imager is an image in whicheach pixel has any one of color information, out of a plurality ofcolors, and the video camera further comprises a convertor forconverting the reduced image extracted by the extractor into an image inwhich each pixel has all the color information of the plurality ofcolors.

Preferably, the video camera further comprises an outputter foroutputting a moving image based on the reduced image extracted by theextractor.

The above described features and advantages of the present inventionwill become more apparent from the following detailed description of theembodiment when taken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a configuration of one embodiment ofthe present invention;

FIG. 2 is an illustrative view showing one example of a mapping state ofan SDRAM applied to the embodiment in FIG. 1;

FIG. 3 is an illustrative view showing one example of an allocationstate of an extraction area and a motion detection area in an imagingsurface;

FIG. 4 is an illustrative view showing one example of an imagestabilizing operation in the embodiment in FIG. 1;

FIG. 5 is a block diagram showing one example of a configuration of amotion detecting circuit applied to the embodiment in FIG. 1;

FIG. 6 is a block diagram showing one example of a configuration of apost-processing circuit applied to the embodiment in FIG. 1;

FIG. 7 is an illustrative view showing one example of an imagingoperation in the embodiment in FIG. 1;

FIG. 8(A) is an illustrative view showing one example of an imagestabilizing process of an extraction area;

FIG. 8(B) is an illustrative view showing another example of the shapechanging process of an extraction area;

FIG. 8(C) is an illustrative view showing still another example of theshape changing process of an extraction area;

FIG. 8(D) is an illustrative view showing yet still another example ofthe shape changing process of an extraction area;

FIG. 9(A) is an illustrative view showing one example of an operationfor determining inclination amounts of a left side and a right side ofan extraction area;

FIG. 9(B) is an illustrative view showing another example of anoperation for determining inclination amounts of a left side and a rightside of an extraction area;

FIG. 10 is an illustrative view showing another example of the imagingoperation in the embodiment in FIG. 1;

FIG. 11(A) is an illustrative view showing one example of an objectscene image inputted to a post-processing circuit;

FIG. 11(B) is an illustrative view showing one example of an objectscene image outputted from the post-processing circuit;

FIG. 11(C) is an illustrative view showing another example of an objectscene image inputted to the post-processing circuit;

FIG. 11(D) is an illustrative view showing another example of an objectscene image outputted from the post-processing circuit;

FIG. 12 is a flowchart showing a part of an operation of a CPU appliedto the embodiment in FIG. 1; and

FIG. 13 is a flowchart showing another portion of the operation of theCPU applied to the embodiment in FIG. 1;

FIG. 14 is a block diagram showing a configuration of another embodimentof the present invention;

FIG. 15(A) is an illustrative view showing one example of a resolutionof an image outputted from an image sensor;

FIG. 15(B) is an illustrative view showing one example of a resolutionof an EIS/AF evaluation image;

FIG. 15(C) is an illustrative view showing one example of a resolutionof an AE/AWB evaluation image;

FIG. 16 is a block diagram showing one example of a configuration of animage sensor applied to the embodiment in FIG. 14;

FIG. 17 is a block diagram showing one example of a configuration of apre-processing circuit applied to the embodiment in FIG. 14;

FIG. 18 is a graph showing one example a zoom magnificationcharacteristic;

FIG. 19(A) is an illustrative view showing one example of an imageoutputted from an image sensor;

FIG. 19(B) is an illustrative view showing one example of an imageoutputted from the pre-processing circuit;

FIG. 19(C) is an illustrative view showing one example of a resolutionof an EIS/AF evaluation image;

FIG. 19(D) is an illustrative view showing one example of a resolutionof an AE/AWB evaluation image;

FIG. 20(A) is an illustrative view showing another example of an imageoutputted from an image sensor;

FIG. 20(B) is an illustrative view showing another example of an imageoutputted from the pre-processing circuit;

FIG. 20(C) is an illustrative view showing another example of aresolution of an EIS/AF evaluation image;

FIG. 20(D) is an illustrative view showing another example of aresolution of an AE/AWB evaluation image;

FIG. 21(A) is an illustrative view showing another example of an imageoutputted from an image sensor;

FIG. 21(B) is an illustrative view showing another example of an imageoutputted from the pre-processing circuit;

FIG. 21(C) is an illustrative view showing another example of aresolution of an EIS/AF evaluation image;

FIG. 21(D) is an illustrative view showing another example of aresolution of an AE/AWB evaluation image;

FIG. 22 is an illustrative view showing one example of an imagingoperation in the embodiment in FIG. 14;

FIG. 23(A) is an illustrative view showing another example of a shapechanging process of an extraction area;

FIG. 23(B) is an illustrative view showing stiff another example of theshape changing process of an extraction area;

FIG. 23(C) is an illustrative view showing yet still another example ofthe shape changing process of an extraction area;

FIG. 23(D) is an illustrative view showing yet still another example ofthe shape changing process of an extraction area;

FIG. 24(A) is an illustrative view showing another example of anoperation for determining inclination amounts of a left side and a rightside of an extraction area;

FIG. 24(B) is an illustrative view showing stiff another example of theoperation for determining inclination amounts of a left side and a rightside of an extraction area;

FIG. 25(A) is an illustrative view showing one example of a shape of anextraction area corresponding to a display magnification of “1.0”;

FIG. 25(B) is an illustrative view showing one example of a shape of anextraction area corresponding to a display magnification of “8.0”;

FIG. 25(C) is an illustrative view showing one example of a shape of anextraction area corresponding to a display magnification of “16”;

FIG. 26 is a graph showing one example of a horizontal margincharacteristic;

FIG. 27 is a flowchart showing another portion of the operation of theCPU applied to the embodiment in FIG. 14;

FIG. 28 is a flowchart showing stiff another portion of the operation ofthe CPU applied to the embodiment in FIG. 14;

FIG. 29 is a flowchart showing yet stiff another portion of theoperation of the CPU applied to the embodiment in FIG. 14;

FIG. 30 is a flowchart showing yet stiff another portion of theoperation of the CPU applied to the embodiment in FIG. 14; and

FIG. 31 is a flowchart showing stiff another portion of the operation ofthe CPU applied to the embodiment in FIG. 14.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

With reference to FIG. 1, a digital video camera 10 of this embodimentincludes an optical lens 12 and an aperture unit 14. An optical image ofan object scene is irradiated onto an imaging surface of a CMOS-typeimage sensor 16 through these members. The imaging surface is coveredwith a primary color filter (not shown) having a Bayer array. Therefore,in each pixel, electric charges having any one of color information,i.e., R (Red), G (Green), and B (Blue), are produced by photoelectricconversion.

When a power supply is inputted, a CPU 28 starts a driver 18 in order toexecute a through-image process. In response to a verticalsynchronization signal Vsync generated at every 1/60 seconds, the driver18 exposes the imaging surface utilizing a focal-plane electronicshutter system and reads out electric charges produced on the imagingsurface in a raster scanning manner. From the image sensor 16, raw imagedata representing an object scene is outputted at a frame rate of 60fps. A pre-processing circuit 20 performs processes, such as a digitalclamp, a pixel defect correction, and a gain control, on the raw imagedata from the image sensor 16, and writes thus processed raw image datato a raw image area 34 a (see FIG. 2) of an SDRAM 34 through a memorycontrol circuit 32.

On the imaging surface, an extraction area EX is allocated in a mannershown in FIG. 3. A post-processing circuit 36 reads out a part of theraw image data belonging to the extraction area EX, out of the raw imagedata accommodated in the raw image area 24 a through the memory controlcircuit 32 at every 1/60 seconds, and performs processes, such as acolor separation, a white balance adjustment, a YUV conversion, and, avertical zoom, on the read-out raw image data. As a result, displayimage data corresponding to a YUV format is created at every 1/60seconds. The created display image data is written into a YUV image area34 b (see FIG. 2) of the SDRAM 34 through the memory control circuit 32.

The LCD driver 38 repeatedly reads out the display image dataaccommodated in the YUV image area 34 b and drives an LCD monitor 40based on the read-out YUV image data. As a result, a real-time movingimage (through image) representing an object scene is displayed on amonitor screen.

The pre-processing circuit 20 executes a simple Y conversion process anda simple RGB conversion process in addition to the above-describedprocesses. The raw image data is converted into Y data by the simple Yconversion process, and converted into RGB data (data in which eachpixel has all color information of R, G and B) by the simple RGBconversion process. The Y data produced by the simple Y conversionprocess is applied to a motion detecting circuit 22 and an AF evaluationcircuit 24, and the RGB data produced by the simple RGB conversionprocess is applied to an AE/AWB evaluation circuit 26.

With reference to FIG. 3, nine motion detection areas MD1 to MD9 areallocated on the imaging surface. The motion detection areas MD1 to MD3form a line in a horizontal direction at an upper level of the imagingsurface, the motion detection areas MD4 to MD6 form a line in ahorizontal direction at a middle level of the imaging surface, andmotion detection areas MD7 to MD9 form a line in a horizontal directionat a lower level of the imaging surface. A minimum number of motiondetection areas to be allocated in a vertical direction is determinedaccording to Equation 1.MN=TM×SF×α  [Equation 1]

-   MN: minimum number of motion detection areas to be allocated in a    vertical direction-   TM: imaging cycle (generation cycle of a vertical synchronization    signal Vsync)-   SF: vibration frequency of the imaging surface-   α: constant (=18)

According to Equation 1, when a multiplied value obtained by multiplyingthe imaging cycle and the vibration frequency of the imaging surface ismultiplied by the constant α, the minimum number of motion detectionareas to be allocated in a vertical direction is obtained. Herein, thevibration frequency of the imaging surface is equivalent to a camerashake frequency (=about 10 Hz) by an operator. Accordingly, when theimaging cycle is “ 1/60 seconds” as in this embodiment, the minimumnumber of motion detection areas to be allocated in a vertical directionbecomes “3”. Furthermore, when the imaging cycle is “ 1/30 seconds”, theminimum number of motion detection areas to be allocated in a verticaldirection becomes “6”.

The motion detecting circuit 22 detects a partial motion vectorrepresenting motion of the object scene in each of the motion detectionareas MD1 to MD9, based on the Y data applied from the pre-processingcircuit 20 at every 1/60 seconds. The motion detecting circuit 22further combines the partial motion vectors of the motion detectionareas MD1 to MD3 to produce a resultant motion vector UVC at every 1/60seconds, combines the partial motion vectors of the motion detectionareas MD4 to MD6 to produce a resultant motion vector MVC at every 1/60seconds, and combines the partial motion vectors of the motion detectionareas MD7 to MD9 to produce a resultant motion vector LVC at every 1/60seconds.

The resultant motion vector UVC represents motion of the object scene atthe upper level of the imaging surface, the resultant motion vector MVCrepresents motion of the object scene at the middle level of the imagingsurface, and the resultant motion vector LVC represents motion of theobject scene at the lower level of the imaging surface.

The CPU 28 creates a total motion vector based on the resultant motionvectors UVC, MVC, and LVC outputted from the motion detecting circuit22, determines whether the motion of the imaging surface in a directionorthogonal to an optical axis is caused due to which of the following:the camera shake or the pan/tilt operation, based on the total motionvector, and moves the extraction area EX along the total motion vectorwhen the motion of the imaging surface is caused due to the camerashake. A position of the extraction area EX is changed so that themotion of the imaging surface caused due to the camera shake iscompensated (offset). When a camera shake occurs on the imaging surface,the extraction area EX moves on the imaging surface in a manner shown inFIG. 4.

The AF evaluation circuit 26 creates an AF evaluation value at every1/60 seconds based on the Y data applied from the pre-processing circuit20. The CPU 28 executes a so-called hill-climbing AF process based onthe created AF evaluation value, and places the optical lens 12 at afocal point.

The AE/AWB evaluation circuit 24 creates an AE/AWB evaluation value atevery 1/60 seconds based on the RGB data applied from the pre-processingcircuit 20. The CPU 28 calculates an EV value capable of obtaining aproper exposure amount and a white balance adjustment gain capable ofobtaining a proper white balance, based on the created AE/AWB evaluationvalue. The CPU 28 further sets an aperture amount and an exposure timethat define the calculated EV value to the aperture unit 14 and thedriver 18, and sets the calculated white balance adjustment gain to thepost-processing circuit 36. As a result, the brightness and the whitebalance of the moving image outputted from the LCD monitor 40 areadjusted moderately.

When a recording start operation is performed by a key input device 30,an I/F 42 is started by the CPU 28. The I/F 42 reads out the image dataaccommodated in the YUV image area 34 b at every 1/60 seconds, andwrites the read-out image data in a moving image file within a recordingmedium 44 in a compressed state. The I/F 42 is stopped by the CPU 28when a recording end operation is performed on the key input device 30.As a result, the recording process of the image data is ended.

The motion detecting circuit 22 is configured as shown in FIG. 5. The Ydata is applied to a frame memory 48 and a distributor 50. The framememory 48 is formed by two banks each having a capacity equivalent toone frame, and the applied Y data is alternately written into the twobanks.

The distributor 50 applies the Y data belonging to the motion detectionareas MD1, MD4, and MD7 to a distributor 52, applies the Y databelonging to the motion detection areas MD2, MD5, and MD8 to adistributor 54, and applies the Y data belonging to the motion detectionareas MD3, MD6, and MD9 to a distributor 56.

The distributor 52 applies the Y data belonging to the motion detectionarea MD1 to a partial-motion-vector detecting circuit 58, applies the Ydata belonging to the motion detection area MD4 to apartial-motion-vector detecting circuit 64, and applies the Y databelonging to the motion detection area MD7 to a partial-motion-vectordetecting circuit 70. The distributor 54 applies the Y data belonging tothe motion detection area MD2 to a partial-motion-vector detectingcircuit 60, applies the Y data belonging to the motion detection areaMD5 to a partial-motion-vector detecting circuit 66, and applies the Ydata belonging to the motion detection area MD8 to apartial-motion-vector detecting circuit 72. The distributor 56 appliesthe Y data belonging to the motion detection area MD3 to apartial-motion-vector detecting circuit 62, applies the Y data belongingto the motion detection area MD6 to a partial-motion-vector detectingcircuit 68, and applies the Y data belonging to the motion detectionarea MD9 to a partial-motion-vector detecting circuit 74.

Each of the partial-motion-vector detecting circuits 58 to 74 comparesthe Y data applied from the distributor 52, 54 or 56 with the Y data ofthe previous frame accommodated in the frame memory 48 so as to detectthe partial motion vector representing the motion of the object scene inthe motion detection area to be noticed. As a result, nine partialmotion vectors respectively corresponding to the motion detection areasMD1 to MD9 are obtained.

A resultant-motion-vector producing circuit 76 combines the threepartial motion vectors respectively detected by thepartial-motion-vector detecting circuits 58, 60, and 62 to produce aresultant motion vector UVC representing the motion of the object sceneat an upper level of the imaging surface. A resultant-motion-vectorproducing circuit 78 combines the three partial motion vectorsrespectively detected by the partial-motion-vector detecting circuits64, 66, and 68 to produce a resultant motion vector MVC representing themotion of the object scene at a middle level of the imaging surface. Aresultant-motion-vector producing circuit 80 combines the three partialmotion vectors respectively detected by the partial-motion-vectordetecting circuits 70, 72 and, 74 to produce a resultant motion vectorLVC representing the motion of the object scene at a lower level of theimaging surface.

The post-processing circuit 36 is configured as shown in FIG. 6. Acontroller 82 repeatedly issues a reading-out request toward the memorycontrol circuit 32 in order to read out the Y data belonging to theextraction area EX from the raw image area 34 a of the SDRAM 34. The rawimage data read out in response thereto undergoes an SRAM 84, and then,is applied to a color separation circuit 86. The color separationcircuit 86 produces RGB data in which each pixel has all colorinformation of R, G and B, based on the applied raw image data.

The produced RGB data is subjected to a white balance adjustment processby a white balance adjustment circuit 88, and then converted into imagedata in a YUV format by a YUV conversion circuit 90. The converted imagedata undergoes a vertical zoom process (described later in detail) by azoom circuit 92, and then, is written into an SRAM 96. A controller 94outputs the image data accumulated in the SRAM 96 to the memory controlcircuit 32, together with the writing request. The outputted image datais written into the YUV image area 34 b of the SDRAM 34 by the memorycontrol circuit 32.

As described above, the image sensor 16 exposes the imaging surfaceutilizing the focal-plane electronic shutter system, and therefore, theexposure timing is different depending on each horizontal pixel column.Then, in the raw image data accommodated in the SDRAM 34, a focal planedistortion in a horizontal direction is generated due to a horizontalmovement of the imaging surface (see FIG. 7), and a focal planedistortion in a vertical direction is generated due to a verticalmovement of the imaging surface (see FIG. 10). Therefore, in thisembodiment, a shape of the extraction area EX is changed based on theoutput from the motion detecting circuit 22 so as to restrain the focusplane distortion.

When the focal plane distortion in a horizontal direction is generated,inclination amounts on a right side and a left side of a rectanglerepresenting the extraction area EX are changed in a manner shown inFIG. 8(A) to FIG. 8(D). That is, when the imaging surface moves in aright direction, the exposure timing of the imaging surface changes in amanner shown in a left column in FIG. 8(A), and therefore, the rightside and the left side of the extraction area EX are inclined in amanner shown in aright column in FIG. 8(A). Furthermore, when theimaging surface moves in a left direction, the exposure timing of theimaging surface changes in a manner shown in a left column in FIG. 8(B),and therefore, the right side and the left side of the extraction areaEX are inclined in a manner shown in a right column of FIG. 8(B).

In addition, when the moving direction of the imaging surface isinverted from right to left, the exposure timing of the imaging surfacechanges in a manner shown in a left column of FIG. 8(C), and therefore,the right side and the left side of the extraction area EX are inclinedin a manner shown in aright column of FIG. 8(C). Furthermore, when themoving direction of the imaging surface is inverted from left to right,the exposure timing of the imaging surface changes in a manner shown ina left column of FIG. 8(D), and therefore, the right side and the leftside of the extraction area EX are inclined in a manner shown in a rightcolumn of FIG. 8(D).

The inclination amount is determined in a manner shown in FIG. 9(A) orFIG. 9(B) based on the resultant vectors UVC, MVC, and LVC outputtedfrom the motion detecting circuit 22. Firstly, horizontal components ofthe resultant motion vectors UVC, MVC, and LVC are specified as “UVCx”,“MVCx” and “LVCx”, respectively. Secondly, a Y coordinate of thehorizontal component UVCx is determined corresponding to verticalpositions of the motion detection areas MD1 to MD3, a Y coordinate ofthe horizontal component MVCx is determined corresponding to verticalpositions of the motion detection areas MD4 to MD6, and a Y coordinateof the horizontal component LVCx is determined corresponding to verticalpositions of the motion detection areas MD7 to MD9.

Moreover, X coordinates at distal ends of the horizontal componentsUVCx, MVCx, and LVCx are determined such that an X coordinate at aproximal end of the horizontal component MVCx is coincident with that ata distal end of the horizontal component UVCx, and an X coordinate at aproximal end of the horizontal component LVCx is coincident with that ata distal end of the horizontal component MVCx. Thereafter, anapproximate function representing a straight line or a curve linking thedetermined three XY coordinates is calculated.

Accordingly, in a case that the horizontal components UVCx, MVCx, andLVCx have a magnitude shown in a left column in FIG. 9(A), three XYcoordinates are determined in a manner shown in a center column in FIG.9(A), and an approximate function having an inclination shown in a rightcolumn in FIG. 9(A) is calculated. Furthermore, when the horizontalcomponents UVCx, MVCx, and LVCx have a magnitude shown in a left columnin FIG. 9(B), three XY coordinates are determined in a manner shown in acenter column in FIG. 9(B), and an approximate function having aninclination shown in a right column in FIG. 9(B) is calculated.

When a focal plane distortion in a vertical direction is generated, avertical size of the extraction area EX is enlarged/reduced in a mannershown in FIG. 11(A) or FIG. 11(C), and a vertical size of the image datacorresponding to the extraction area EX is restored to the original sizeby the zoom circuit 92 arranged in the post-processing circuit 36 in amanner shown in FIG. 11(B) or FIG. 11(D). A change magnification of thevertical size of the extraction area EX is calculated according toEquation 2, and the zoom magnification of the zoom circuit 92 iscalculated according to Equation 3.ZM1=1+VM/VPX  [Equation 2]

-   ZM1: a change magnification of the vertical size of the extraction    area EX-   VM: the number of pixels equivalent to the vertical component of the    total motion vector-   VPX: the number of vertical pixels of the imaging surface    ZM2=1/ZM1  [Equation 3]-   ZM2: zoom magnification

According to Equation 2, a numerical value obtained by adding “1” to aratio of the number of pixels equivalent to the vertical component ofthe total motion vector to the number of vertical pixels of the imagingsurface is equivalent to a change magnification of the vertical size ofthe extraction area EX. According to Equation 3, the zoom magnificationof the zoom circuit 92 is equivalent to an inverse number of the changemagnification of the vertical size of the extraction area EX.

The CPU 28 processes a plurality of tasks, including an imagestabilizing task shown in FIG. 12 to FIG. 13, in parallel. It is notedthat control programs corresponding to these tasks are stored in a flashmemory 46.

With reference to FIG. 12, it is determined whether or not a verticalsynchronization signal Vsync is generated in a step S1, and if YES, theresultant motion vectors UVC, MVC, and LVC are fetched from the motiondetecting circuit 22 in a step S3. In a step S5, the total motion vectoris created based on the fetched resultant motion vectors UVC, MVC, andLVC. In a step S7, an area-shape changing process is executed, and in asucceeding step S9, it is determined whether or not the motion of theimaging surface at the current time point is caused due to a pan/tiltoperation based on the total motion vector. If YES in this step, theprocess directly returns to the step S1, and if NO, the extraction areaEX is moved along the total motion vector in a step S11, and then, theprocess returns to the step S1.

The area-shape changing process in the step S7 is executed according toa subroutine shown in FIG. 13. Firstly, in a step S21, the horizontalcomponents of the resultant motion vectors UVC, MVC, and LVC arespecified as “UVCx”, “MVCx” and “LVCx”. In a step S23, a Y coordinate ofthe horizontal component UVCx is determined corresponding to verticalpositions of the motion detection areas MD1 to MD3, and a Y coordinateof the horizontal component MVCx is determined corresponding to verticalpositions of the motion detection areas MD4 to MD6, and a Y coordinateof the horizontal component LVCx is determined corresponding to verticalpositions of the motion detection areas MD7 to MD9.

In a step S25, the X coordinates at the distal end of the horizontalcomponents UVCx, MVCx and LVCx are determined such that the X coordinateat the proximal end of the horizontal component MVCx is coincident withthe X coordinate at the distal end of the horizontal component UVCx, andthe X coordinate at the proximal end of the horizontal component LVCx iscoincident with the X coordinate at the distal end of the horizontalcomponent MVCx. In a step S27, an approximate function representing astraight line or a curve linking the determined three XY coordinates iscalculated, and in a step S29, inclination amounts of a right side and aleft side of the extraction area EX are determined according to thecalculated approximate function.

In a step S31, the vertical component of the total motion vector isspecified. In a step S33, a change magnification of a vertical size ofthe extraction area EX is calculated according to the Equation 2, and ina step S35, a zoom magnification of the zoom circuit 92 is calculatedaccording to the Equation 3. Upon completion of the process in the stepS35, the process is restored to a routine at a hierarchical upper level.

As understood from the above description, the image sensor 16 repeatedlyoutputs in a raster scanning manner an object scene image produced onthe imaging surface by the exposure operation of the focal-planeelectronic shutter system. The post-processing circuit 36 extracts apartial object scene image belonging to the extraction area EX, out ofthe object scene image outputted from the image sensor 16. A movingimage based on the extracted partial object scene image is displayed onthe LCD monitor 40 by the LCD driver 38. The motion detecting circuit 22detects motion of the imaging surface in a direction orthogonal to theoptical axis. The position of the extraction area EX is changed by theCPU 28 such that the motion detected by the motion detecting circuit 22is compensated (S11). The CPU 28 further changes the shape of theextraction area EX based on the motion detected by the motion detectingcircuit 22 so that a focal plane distortion is restrained (S7).

Thus, the position of the extraction area EX is changed so that themotion of the imaging surface in a direction orthogonal to the opticalaxis is compensated, and the shape of the extraction area EX is changedso that the focal plane distortion is restrained. As a result, the shakeof the imaging surface and the focal plane distortion are correctedunitarily, and therefore, it is possible to ameliorate the quality ofthe moving image with a simple circuit configuration.

It is noted that in this embodiment, the extraction area EX is movedalong the total motion vector. However, in addition thereto, theevaluation area referenced by the AE/AWB evaluation circuit 26 may bemoved as if to follow the extraction area EX.

With reference to FIG. 14, a digital video camera 110 according to thisembodiment includes a zoom lens 112, a focus lens 114, and an apertureunit 116 respectively driven by drivers 120 a, 120 b, and 120 c. Anoptical image of an object scene is irradiated onto an imaging surfaceof a CMOS-type image sensor 118 through these members. The imagingsurface has an effective pixel area equivalent to horizontal 3072pixels×vertical 1728 pixels, and is also covered with a primary colorfilter (not shown) having a Bayer array Electric charges produced ineach pixel have any one of color information, i.e., R (Red), G (Green),and B (Blue).

When a power source is inputted, a CPU 136 applies a correspondingcommand to the driver 120 d in order to execute a through-image process.From an SG (Signal Generator) 122, a vertical synchronization signalVsync is generated at every 1/30 seconds, for example. The driver 120 dexposes the imaging surface according to a focal-plane electronicshutter system in response to the vertical synchronization signal Vsyncgenerated from the SG 122, and reads out the electric charges producedthereby from the imaging surface. The image sensor 118 has N (N: integerof equal to or more than 2, “4”, for example) of channels CH1 to CHN,and raw image data based on the read-out electric charges are outputteddispersively (in parallel) from the channels CH1 to CHN. The outputtedraw image data has a resolution of horizontal 3072 pixels×vertical 1728pixels, as shown in FIG. 15(A).

The pre-processing circuit 124 respectively performs pre-processes ofparallel N systems on the raw image data of N channels outputted fromthe image sensor 118. The pre-processes of each system are configured bya noise removal, a reduction zoom, and an edge adjustment, and the rawimage data that undergoes such the pre-processes is written in a rawimage area 142 a of an SDRAM 142 through a memory control circuit 140.

It is noted that the reduction zoom in the pre-processing circuit 124 isexecuted by a zoom circuit 124 z. Below, the reduction zoom executed bythe zoom circuit 124 z is defined as “RAW zoom”.

The raw image data (resolution: horizontal 3072 pixels×vertical 1728pixels) on which the noise removal is performed by the pre-processingcircuit 122 is applied to evaluation-image creating circuits 126 and128. The evaluation-image creating circuit 126 performs an addingprocess of vertical two pixels and an adding process of horizontal twopixels on the applied raw image data so as to create EIS/AF evaluationimage data. On the other hand, the evaluation-image creating circuit 128performs an adding process of horizontal four pixels on the applied rawimage data so as to create AE/AWB evaluation image data.

The EIS/AF evaluation image data has a resolution of horizontal 1536pixels×vertical 864 pixels, as shown in FIG. 15(B). The AE/AWBevaluation image data has a resolution of horizontal 768 pixels×vertical1728 pixels, as shown in FIG. 15(C). The EIS/AF evaluation image data isapplied to the motion detecting circuit 130 and the AF evaluationcircuit 132, and the AE/AWB evaluation image data is applied to theAE/AWB evaluation circuit 134.

With reference to FIG. 15(A) and FIG. 15(B), one extraction area EX andnine motion detection areas MD11 to MD19 are allocated to the imagingsurface. The extraction area EX is a rectangular area having a size ofhorizontal 1920 pixels×vertical 1080 pixels. Furthermore, the motiondetection areas MD11 to MD13 form a line in a horizontal direction at anupper level of the imaging surface, the motion detection areas MD14 toMD16 form a line in a horizontal direction at a middle level of theimaging surface, and the motion detection areas MD17 to MD19 form a linein a horizontal direction at a lower level of the imaging surface.

The motion detecting circuit 130 detects a partial motion vectorrepresenting motion of an object scene in each of the motion detectionareas MD11 to MD19, based on the EIS/AF evaluation image data. Themotion detecting circuit 130 also combines partial motion vectors in themotion detection areas MD11 to MD13 so as to produce a resultant motionvector UVC, combines partial motion vectors in the motion detectionareas MD14 to MD16 to produce a resultant motion vector MVC, andcombines partial motion vectors in the motion detection areas MD17 toMD19 so as to produce a resultant motion vector LVC.

Either a detection process for the partial motion vector or a creatingprocess for the resultant motion vector is executed at each generationof the vertical synchronization signal Vsync. Furthermore, the resultantmotion vector UVC represents motion of an object scene at the upperlevel of the imaging surface, the resultant motion vector MVC representsmotion of an object scene at the middle level of the imaging surface,and the resultant motion vector LVC represents motion of an object sceneat the lower level of the imaging surface.

The CPU 136 creates a total motion vector based on the resultant motionvectors UVC, MVC, and LVC outputted from the motion detecting circuit130, determines whether the motion of the imaging surface in a directionorthogonal to an optical axis is caused due to which of the following:the camera shake or the pan/tilt operation, based on the total motionvector, and moves the extraction area EX along the total motion vectorwhen the motion of the imaging surface is caused due to the camerashake. A position of the extraction area EX is changed so that themotion of the imaging surface is caused due to the camera shake iscompensated (offset).

A post-processing circuit 144 reads out the partial raw image databelonging to the extraction area EX, out of the raw image dataaccommodated in the raw image area 142 a, through the memory controlcircuit 140, and performs post processes, such as a color separation, awhite balance adjustment, a YUV conversion, and an enlargement zoom, onthe read-out partial raw image data. The partial raw image data is readout from the raw image area 142 a in response to the verticalsynchronization signal Vsync, and the post-process is also executed inresponse to the vertical synchronization signal Vsync. The image data ofa YUV format thus produced is outputted from a moving-image outputterminal M_OUT, and written into a moving image area 142 b of the SDRAM142 through the memory control circuit 140.

It is noted that each of a plurality of pixels forming the image data onwhich the color separation process is performed has all colorinformation of R, Q and B. The format of such image data is converted toa YUV format by the YUV conversion, and the enlargement zoom is furtherperformed thereon. In addition, the enlargement zoom in thepost-processing circuit 144 is executed by the zoom circuit 144 z.Below, the enlargement zoom executed by the post-processing circuit 144is defined as “YUV zoom”.

The LCD driver 146 repeatedly reads out the image data accommodated inthe moving image area 142 b, and drives an LCD monitor 148 based on theread-out image data. As a result, a real-time moving image (throughimage) representing an object scene is displayed on a monitor screen.

The AE/AWB evaluation circuit 134 integrates a part of the AE/AWBevaluation image data belonging to a photometric/white balance area EWAshown in FIG. 15(C), out of the AE/AWB evaluation image data outputtedfrom the evaluation-image creating circuit 128, at each generation ofthe vertical synchronization signal Vsync, and outputs an integralvalue, i.e., an AE/AWB evaluation value. The CPU 136 executes an AE/AWBprocess in order to calculate a proper EV value and a proper whitebalance adjustment gain based on the AE/AWB evaluation value outputtedfrom the AE/AWB evaluation circuit 134. An aperture amount and anexposure time that define the calculated proper EV value are set to thedrivers 120 c and 120 d, respectively, and the calculated proper whitebalance adjustment gain is set to the post-processing circuit 144. As aresult, the brightness and the white balance of the moving imageoutputted from the LCD monitor 148 is adjusted moderately.

The AF evaluation circuit 132 extracts a part of the EIS/AF evaluationimage data belonging to a focus area FA shown in FIG. 15(B), out of theEIS/AF evaluation image data outputted from the evaluation-imagecreating circuit 126, and integrates a high-frequency component of theextracted EIS/AF evaluation image data in response to the verticalsynchronization signal Vsync. The calculated integral value, i.e., theAF evaluation value, is applied to the CPU 136 for a continuous AFprocess. The CPU 136 references the applied AF evaluation value so as tocontinually search a focal point by a so-called hill-climbing process.The focus lens 114 is placed at the discovered focal point.

When the zoom button 138 z on the key input device 138 is operated, theCPU 136 sets a display magnification different by a predetermined amount(minute amount) in a desired direction than a display magnification atthe current time point, as a target display magnification, andcalculates an optical zoom magnification, a RAW zoom magnification, anda YUV magnification, corresponding to the set target displaymagnification.

Subsequent thereto, the CPU 136 sets the calculated optical zoommagnification, RAW zoom magnification, and YUV zoom magnification, tothe driver 120 a, the zoom circuit 124 z, and the zoom circuit 144 z,respectively, in order to execute the zoom process. Thereby, a throughimage having the target display magnification is outputted from the LCDmonitor 148.

Thereafter, the CPU 136 changes settings of the motion detection areasMD11 to MD19, the focus area FA, and the photometric/white balance areaEWA so as to be adapted to the RAW zoom magnification set to the zoomcircuit 124 z and the YUV zoom magnification set to the zoom circuit 144z. This improves the accuracy of the image stabilizing process, thecontinuous AF process, and the AE/AWB process.

When a movie button 138 m on the key input device 138 is operated, theCPU 136 applies a recording start command to an I/F 150 to start amoving-image recording process. The I/F 150 creates a moving image filewithin a recording medium 152, cyclically reads out the image dataaccommodated in the moving image area 142 b, and writes the read-outimage data into the moving image file within the recording medium 152.When the movie button 138 m is operated again, a recording stop commandis applied to the I/F 150. The I/F 150 ends reading-out of the imagedata from the moving image area 142 b, and closes the moving image fileof a write destination. Thereby, the moving image file is completed.

When the shutter button 138 s on the key input device 138 is operatedwhile the moving-image recording process is being executed, the CPU 136applies a still-image extraction command to the post-processing circuit144 in order to execute a parallel-still-image recording process, andalso applies a still-image recording command to the I/F 150. Thepost-processing circuit 144 outputs one frame of image data representingan object scene image at a time point when the shutter button 138 s isoperated, from a still-image output terminal S_OUT. The outputted imagedata is written into the still image area 142 b of the SDRAM 142 throughthe memory control circuit 140. The I/F 150 reads out the image dataaccommodated in the still image area 142 c through the memory controlcircuit 140, and creates a stiff image file containing the read-outimage data within the recording medium 152.

On the other hand, when the shutter button 138 s is operated in a statethat the moving-image recording process is interrupted, the CPU 136, inorder to execute an independent still-image recording process, sets theRAW zoom magnification and the YUV zoom magnification, both indicating“1.0”, to the zoom circuits 124 z and 144 z and applies the stiff-imageprocessing command and the still-image recording command to thepreprocessing circuit 124, the post-processing circuit 144, and the I/F150.

Thereby, one frame of raw image data having a resolution of horizontal3072 pixels×vertical 1728 pixels is outputted from the pre-processingcircuit 124, and written into the raw image area 142 a of the SDRAM 142.

The post-processing circuit 144 reads out the raw image data having thesame resolution from the raw image area 142 a, and outputs image data ofa YUV format based on the read-out raw image data, from the still-imageoutput terminal S_OUTS The outputted image data is written into thestill image area 142 c of the SDRAM 142 through the memory controlcircuit 140.

The I/F 150 reads out the image data accommodated in the still imagearea 142 c through the memory control circuit 140, and creates a stiffimage file containing the read-out image data within the recordingmedium 152. Upon completion of the recording, the above-mentionedthrough-image process is resumed.

The image sensor 118 is configured as shown in FIG. 16. The electriccharges representing the object scene image is produced by a pluralityof light-receiving elements 156, 156, . . . , placed in a matrix. Eachlight-receiving element 156 is equivalent to the above-mentioned pixel.Each light-receiving element 156, 156, . . . , forming a line in avertical direction is connected to a common CDS circuit 162 via an A/Dconverter 158 and a row selection switch 160. The electric chargesproduced in the light-receiving element 156 are converted to 12-bitdigital data by the A/D converter 158. A vertical scanning circuit 166executes an operation for turning on/off the row selection switch 160,160, . . . , in a raster scanning manner for each pixel, in order toexpose the imaging surface in a focal-plane electronic shutter system.The noise included in the pixel data that has undergone the rowselection switch 160 in an on state is removed by the CDS circuit 162.

A column selection switch 1641 is allocated to the CDS circuit 162 at anN*M+1-th column (M: 0, 1, 2, 3, . . . ), and a column selection switch1642 is allocated to the CDS circuit 162 at an N*M+2-th column.Similarly, a column selection switch 164N is allocated to the CDScircuit 162 at an N*M+N-th column.

A horizontal scanning circuit 168 turns on the column selection switch1641 at a timing at which the row selection switch 160 at the N*M+1-thcolumn is turned on, and turns on the column selection switch 1642 at atiming at which the row selection switch 160 at the N*M+2-th column isturned on. Similarly, the horizontal scanning circuit 168 turns on thecolumn selection switch 164N at a timing at which the row selectionswitch 160 at the N*M+N-th column is turned on.

As a result, partial raw image data based on the electric chargesproduced in the light-receiving element 156 at the N*M+1-th column isoutputted from the channel CH1, and partial raw image data based on theelectric charges produced in the light-receiving element 156 at theN*M+2-th column is outputted from the channel CH2. Partial raw imagedata based on the electric charges produced in the light-receivingelement 156 at the N*M+N-th column is outputted from the channel CHN.

The pre-processing circuit 124 is configured as shown in FIG. 17. Thepartial raw mage data of the channel CH1 is applied to thepre-processing block PB1, and the partial raw image data of the channelCH2 is applied to the pre-processing block PB2. The partial raw magedata of the channel CHN is applied to the pre-processing block PBN.

The pre-processing block PB1 is configured by an LPF 1701, a reductionzoom circuit 1721, and an edge adjustment circuit 1741. Thepre-processing block PB2 is configured by an LPF 1702, a reduction zoomcircuit 1722, and an edge adjustment circuit 1742. The pre-processingblock PBN is configured by an LPF 170N, a reduction zoom circuit 172N,and an edge adjustment circuit 174N. It is noted that the zoom circuit124 z shown in FIG. 14 is configured by the reduction zoom circuits 1721to 172N.

Therefore, the partial raw image data of each channel is subjected to aseries of processes of a noise removal, a reduction zoom, and an edgeadjustment, in parallel to one another. The partial raw image data onwhich the noise removal is performed is outputted toward theevaluation-image creating circuits 126 and 128, while the partial rawimage data on which the edge adjustment is performed is written into theSDRAM 178. A controller 176 issues a write request toward the memorycontrol circuit 140 each time an amount of data accommodated in theSDRAM 178 reaches a threshold value, and when an acknowledge signal issent back from an issuing destination, outputs a predetermined amount ofraw image data toward the memory control circuit 140.

A process for setting a zoom magnification in response to the operationof the zoom button 138 z and a process for setting the motion detectionareas MD11 to MD19, the focus area FA, and the photometric/white balancearea EWA with reference to the RAW zoom magnification are executed in amanner described below. When a target display magnification is set, theoptical zoom magnification, the RAW zoom magnification, and the YUV zoommagnification are calculated with reference to a graph shown in FIG. 18.It is noted that data equivalent to the graph shown in FIG. 18 is savedin a flash memory 154 as graph data GRD1.

According to FIG. 18, the optical zoom magnification indicates “1.0”when the zoom lens 112 is positioned at a wide end, and indicates “10.0”when the zoom lens 112 is positioned at a tele end. Furthermore, theoptical zoom magnification increases linearly as the zoom lens 112 movesfrom the wide end to the tele end, and is maintained at “10.0” in arange that the display magnification exceeds “16”. The YUV zoommagnification is maintained at “1.0” in a range that the displaymagnification is equal to or less than “16”, and increases linearly upto “10.0” in a range that the display magnification exceeds “16”.

The RAW zoom magnification indicates “0.625” corresponding to thedisplay magnification=1.0 (zoom lens 12=wide end), and indicates “1.0”corresponding to the display magnification=16 (zoom lens 12=tele end).Furthermore, the RAW zoom magnification increases linearly as thedisplay magnification moves from “1.0” toward “16”, and is maintained at“1.0” in a range that the display magnification exceeds “16”.

When a target display magnification is set to “1.0”, “1.0” is calculatedas the optical zoom magnification, “0.625” is calculated as the RAW zoommagnification, and “1.0” is calculated as the YUV zoom magnification.Furthermore, when the target display magnification is set to “8.0”,“5.0” is calculated as the optical zoom magnification, “0.7692” iscalculated as the RAW zoom magnification, and “1.0” is calculated as theYUV zoom magnification. Also, when the target display magnification isset to “16”, “10.0” is calculated as the optical zoom magnification,“1.0” is calculated as the RAW zoom magnification, and “1.0” iscalculated as the YUV zoom magnification.

The optical zoom magnification, the RAW zoom magnification, and the YUVzoom magnification thus calculated are set to the driver 120 a, the zoomcircuit 124 z, and the zoom circuit 144 z, respectively. Furthermore,the motion detection areas MD11 to MD19, the focus area FA, and thephotometric/white balance area EWA are allocated to the imaging surfacein a mode different depending on the value of the set RAW zoommagnification.

When the raw image data shown in FIG. 19(A) is outputted from the imagesensor 118 corresponding to the optical zoom magnification “1.0”, rawimage data having a size as shown in FIG. 19(B) (=horizontal 1935pixels×vertical 1088 pixels) is outputted from the pre-processingcircuit 124. The post-processing circuit 144 performs a post process ona part of the raw image data belonging to the extraction area EX (size:horizontal 1920 pixels×vertical 1080 pixels), out of the raw image datashown in FIG. 19(B). Because the YUV zoom magnification is “1.0”, animage of an angle of view equivalent to the extraction area EX isdisplayed on the LCD monitor 148.

Furthermore, as shown in FIG. 19(C), the focus area FA is allocated tothe entire region of the EIS/AF evaluation image, and the motiondetection areas MD11 to MD19 are allocated on the EIS/AF evaluationimage so as to establish a predetermined relationship with the focusareas FA. Furthermore, as shown in FIG. 19(D), the photometric/whitebalance area EWA is allocated to the entire area of the AE/AWBevaluation image.

When the optical zoom magnification is changed to “5.0”, the raw imagedata shown in FIG. 20(A) is outputted from the image sensor 118. Becausethe RAW zoom magnification is changed to “0.7692”, raw image data havinga size as shown in FIG. 20(B) (=horizontal 2363 pixels×vertical 1329pixels) is outputted from the pre-processing circuit 124. Thepost-processing circuit 144 performs a post process on a part of the rawimage data belonging to the extraction area EX, out of the raw imagedata shown in FIG. 20(B). The YUV zoom magnification is “1.0”, and as aresult, a through image of an angle of view equivalent to the extractionarea EX shown in FIG. 20(B) is displayed on the LCD monitor 148.

Furthermore, with reference to FIG. 20(C), a focus area FA having a sizeequivalent to horizontal 1258 pixels×vertical 697 pixels is allocated inthe center of the EIS/AF evaluation image. The motion detection areasMD11 to MD19 are allocated on the EIS/AF evaluation image so as toestablish a predetermined relationship with the focus areas FA.Furthermore, with reference to FIG. 20(D), the photometric/white balancearea EWA, which has horizontal 590 pixels×vertical 1329 pixels, isallocated on the AE/AWB evaluation image.

When the optical zoom magnification is changed to “10.0”, raw image datashown in FIG. 21(A) is outputted from the image sensor 118. The RAW zoommagnification is changed to “1.0”, and raw image data having a size asshown in FIG. 21(B) (=horizontal 3096 pixels×vertical 1728 pixels) isoutputted from the pre-processing circuit 124. The post-processingcircuit 144 performs a post process on a part of the raw image databelonging to the extraction area EX, out of the raw image data shown inFIG. 21(B). The YUV zoom magnification is “1.0”, and as a result, athrough image of an angle of view equivalent to the extraction area EXshown in FIG. 21(B) is displayed on the LCD monitor 148.

With reference to FIG. 21(C), a focus area FA having a size equivalentto horizontal 968 pixels×vertical 540 pixels is allocated in the centerof the EIS/AF evaluation image. The motion detection areas MD11 to MD 19are allocated on the EIS/AF evaluation image so as to establish apredetermined relationship with such a focus areas FA. Furthermore, withreference to FIG. 21(D), the photometric/white balance area EWA, whichhas horizontal 484 pixels×vertical 1080 pixels, is allocated on theAE/AWB evaluation image.

Thus, the RAW zoom magnification increases as the optical zoommagnification increases, and decreases as the optical zoom magnificationdecreases. Therefore, the angle of view of the object scene image basedon the raw image data extracted by the post-processing circuit 144 isdecreased by a rate exceeding a decrease rate caused due to the increaseof the optical zoom magnification, and is increased by a rate exceedingan increase rate caused due to the decrease of the optical zoommagnification. As a result, in a low zoom magnification range, it ispossible to secure a wide angle of view irrespective of the increase inthe resolution of the imaging surface. Also, in a high zoommagnification range, a zoom effect is increased. Thus, it is possible toimprove the reproducing performance of the object scene image.

As described above, because the image sensor 116 exposes the imagingsurface in a focal-plane electronic shutter system, the exposure timingvaries depending on each horizontal pixel column. Then, due to thehorizontal movement of the imaging surface, the focal plane distortionin a horizontal direction is generated in the raw image dataaccommodated in the raw image area 142 a (See FIG. 22). Thus, the CPU136 changes the shape of the extraction area EX so as to restrain thefocus plane distortion, based on the resultant vectors UVC, MVC, and LVCfetched from the motion detecting circuit 130.

When the focal plane distortion is generated in a horizontal direction,the inclination amount of the right side and the left side of therectangle representing the extraction area EX is changed in a mannershown in FIG. 23(A) to FIG. 23(D). That is, when the imaging surfacemoves in a right direction, the exposure timing of the imaging surfacechanges in a manner shown in a left column of FIG. 23(A), and therefore,the right side and the left side of the extraction area EX are inclinedin a manner shown in a right column of FIG. 23(A). Also, when theimaging surface moves in a left direction, the exposure timing of theimaging surface changes in a manner shown in a left column of FIG.23(B), and therefore, the right side and the left side of the extractionarea EX are inclined in a manner shown in a right column of FIG. 23(B).

Furthermore, when a moving direction of the imaging surface is invertedfrom right to left, the exposure timing of the imaging surface ischanged in a manner shown in a left column of FIG. 23(C), and therefore,the right side and the left side of the extraction area EX are inclinedin a manner shown in a right column of FIG. 23(C). Furthermore, when themoving direction of the imaging surface is inverted from left to right,the exposure timing of the imaging surface is changed in a manner shownin a left column of FIG. 23(D), and therefore, the right side and theleft side of the extraction area EX are inclined in a manner shown in aright column of FIG. 23(D).

The inclination amount is determined in a manner shown in FIG. 24(A) orFIG. 24(B), based on the resultant vectors UVC, MVC, and LVC outputtedfrom the motion detecting circuit 122. Firstly, the horizontalcomponents of the resultant motion vectors UVC, MVC, and LVC arespecified as “UVCx”, “MVCx”, and “LVCx”. Subsequently, a Y coordinate ofthe horizontal component UVCx is determined corresponding to verticalpositions of the motion detection areas MD11 to MD13, a Y coordinate ofthe horizontal component MVCx is determined corresponding to verticalpositions of the motion detection areas MD14 to MD16, and a Y coordinateof the horizontal component LVCx is determined corresponding to verticalpositions of the motion detection areas MD17 to MD19.

Moreover, X coordinates at distal ends of the horizontal componentsUVCx, MVCx, and LVCx are determined such that an X coordinate at aproximal end of the horizontal component MVCx is coincident with that ata distal end of the horizontal component UVCx, and an X coordinate at aproximal end of the horizontal component LVCx is coincident with that ata distal end of the horizontal component MVCx. Thereafter, anapproximate function representing a straight line or a curve linking thedetermined three XY coordinates is calculated.

Therefore, when the horizontal components UVCx, MVCx, and LVCx have asize shown in a left column of FIG. 24(A), the three XY coordinates aredetermined in a manner shown in a center column of FIG. 24(A), and anapproximate function having an inclination shown in a right column ofFIG. 24(A) is calculated. Furthermore, when the horizontal componentsUVCx, MVCx, and LVCx have a size shown in a left column of FIG. 24(B),three XY coordinates are determined in a manner shown in a center columnof FIG. 24(B), and an approximate function having an inclination shownin a right column of FIG. 24(B) is calculated. The shape of theextraction area EX is changed with reference to the approximatefunctions thus calculated.

When a camera shake occurs, the imaging surface vibrates by about amaximum of 10 Hz. Then, in the extraction area EX, an inclinationequivalent to a maximum of horizontal five pixels is generatedcorresponding to a display magnification of “1.0”, an inclinationequivalent to a maximum of horizontal 40 pixels is generatedcorresponding to a display magnification of “8.0”, and an inclinationequivalent to a maximum of horizontal 80 pixels is generatedcorresponding to a display magnification of “16” (see FIG. 25(A) to FIG.25(C)).

On the other hand, although a size of the raw image data is changed inresponse to the operation of the zoom button 138 z, the size exceeds thesize of the extraction area EX in the entire region of the zoom range.In particular, when the display magnification of “1.0” is set, a marginof 15 pixels is secured in a horizontal direction, as shown in FIG.19(B). As a result, it becomes also possible to correct the focal planedistortion even by the minimum display magnification.

However, a movable range of the extraction area EX decreasescorresponding to such deformation of the extraction area EX. That is,the movable range of the extraction area EX decreases by a maximum ofhorizontal 15 pixels corresponding to the display magnification of“1.0”, decreases by a maximum of horizontal 42 pixels corresponding to adisplay magnification of “8.0”, and decreases by a maximum of horizontal64 pixels corresponding to a display magnification of “16”.

Therefore, with reference to a graph shown in FIG. 26, the CPU 136specifies a horizontal margin corresponding to the target displaymagnification so as to determine a moving amount of the extraction areaEX with reference to the specified horizontal margin. The moving amountof the extraction area EX is equivalent to an amount obtained bydeducting the horizontal margin from the horizontal component of thetotal motion vector. Thereby, it becomes possible to avoid a situationwhere a part of the extraction area EX swerves from the raw image area142 a. It is noted that data corresponding to values in the graph shownin FIG. 26 are saved in the flash memory 154 as graph data GRD2.

The CPU 136 executes in parallel a plurality of tasks including animaging task shown in FIG. 27, an image stabilizing task shown in FIG.28 and FIG. 29, and a zoom control task shown in FIGS. 30 and 31. It isnoted that control programs corresponding to these tasks are stored inthe flash memory 154.

With reference to FIG. 27, a through-image process is started in a stepS101, and a continuous AF process is started in a step S103. As a resultof the process in the step S101, raw image data having a resolution ofhorizontal 3072 pixels×vertical 1928 pixels is outputted from the imagesensor 118 at every 1/30 seconds, and a through image based on this rawimage data is outputted from the LCD monitor 148. Also, as a result ofthe process in the step S103, the position of the focus lens 114 isadjusted continuously.

In a step S105, the AE/AWB process is executed. As a result, thebrightness and the white balance of the through image are adjustedmoderately. In a step S107, it is determined whether or not the moviebutton 38 m is operated, and in a step S109, it is determined whether ornot the shutter button 138 s is operated.

When the movie button 138 m is operated, the process proceeds to a stepS111 from the step S107 so as to determine whether or not themoving-image recording process is being executed. When NO is determinedin this step, the moving-image recording process is started in a stepS113, and on the other hand, when YES is determined, the moving-imagerecording process is stopped in a step S115. Upon completion of theprocess in the step S113 or S115, the process returns to the step S105.When the shutter button 138 s is operated, the independent still-imagerecording process or the parallel still-image recording process isexecuted in a step S117, and thereafter, the process returns to the stepS105.

With reference to FIG. 28, it is determined whether or not the verticalsynchronization signal Vsync is generated in a step S121, and when YESis determined, the resultant motion vectors UVC, MVC, and LVC arefetched from the motion detecting circuit 130 in a step S123. In a stepS125, the total motion vector is created based on the fetched resultantmotion vectors UVC, MVC, and LVC. In a step S127, the area-shapechanging process is executed, and in a subsequent step S129, it isdetermined whether or not the motion of the imaging surface at thecurrent time point is caused due to the pan/tilt operation based on thetotal motion vector. When YES is determined in this step, the processreturns as it is to the step S101. When NO is determined, the processproceeds to a step S131 so as to calculate the moving amount of theextraction area EX with reference to the total motion vector created inthe step S125 and a horizontal margin specified in a step S163 describedlater. The moving amount is equivalent to an amount obtained bydeducting the horizontal margin from the horizontal component of thetotal motion vector. In a step S133, the extraction area EX is movedaccording to the calculated moving amount, and thereafter, the processreturns to the step S121.

The area-shape changing process in the step S127 is executed accordingto a sub-routine shown in FIG. 29. Firstly, in a step S141, thehorizontal components of the resultant motion vectors UVC, MVC, and LVCare specified as “LCx”, “MVCx”, and “LVCx”. In a step S143, a Ycoordinate of the horizontal component UVCx is determined correspondingto vertical positions of the motion detection areas MD11 to MD13, a Ycoordinate of the horizontal component MVCx is determined correspondingto vertical positions of the motion detection areas MD14 to MD16, and aY coordinate of the horizontal component LVCx is determinedcorresponding to vertical positions of the motion detection areas MD17to MD19.

In a step S145, X coordinates at distal ends of the horizontalcomponents UVCx, MVCx, and LVCx are determined such that an X coordinateat the proximal end of the horizontal component MVCx is coincident withthat at the distal end of the horizontal component UVCx, and an Xcoordinate at the proximal end of the horizontal component LVCx iscoincident with that at the distal end of the horizontal component MVCx.In a step S147, an approximate function that represents a straight lineor a curve linking the determined three XY coordinates is calculated,and in a step S149, inclination amounts of the right side and the leftside of the extraction area EX are determined according to thecalculated approximate function. Upon completion of the process in thestep S149, the process is restored to a routine at a hierarchical upperlevel.

With reference to FIG. 30, zoom settings are initialized in a step S151,and in a step S153, it is determined whether or not the zoom button 138z is operated. When a determination result is updated from NO to YES,the process proceeds to a step S155 so as to set a display magnificationdifferent depending on each operation mode of the zoom button 138 z asthe target display magnification. In a step S157, with reference to thegraph shown in FIG. 18, an optical zoom magnification, a RAW zoommagnification, and a YUV zoom magnification, each of which correspondsto the target display magnification, are calculated.

In a step S159, in order to execute the zoom process, the calculatedoptical zoom magnification, RAW zoom magnification, and YUV zoommagnification are set to the driver 120 a, the zoom circuit 124 z, andthe zoom circuit 144 z, respectively. Thereby, a through image havingthe target display magnification is outputted from the LCD monitor 148.

In a step S161, settings of the motion detection areas MD11 to MD19, thefocus area FA, and the photometric/white balance area EWA are changed tobe adapted to the RAW zoom magnification set in the step S159. As aresult, the image stabilizing process, the continuous AF process, andthe AE/AWB process are executed highly accurately. In the step S163, thehorizontal margin corresponding to the target display magnification isspecified with reference to the graph shown in FIG. 26. Upon completionof the process in the step S163, the process returns to the step S153.

The zoom process in the step S159 is executed according to a sub-routineshown in FIG. 31. Firstly, it is determined in a step S171 whether ornot both the current display magnification and the target displaymagnification are within a range between 1.0 times to 16 times, andthen, it is determined in a step S173 whether or not both the currentdisplay magnification and the target display magnification are in arange exceeding 16 times.

When YES is determined in the step S171, the optical zoom magnificationis changed in a step S175. Upon completion of the changing operation ofthe optical zoom magnification, YES is determined in a step S177, andthe RAW zoom magnification is changed in a step S179. When YES isdetermined in the step S173, the YUV zoom magnification is changed in astep S181. When NO is determined in the step S173, the correspondingmagnification changing process is executed in a step S183, regardingthat both the current display magnification and the target displaymagnification exceed 16 times. Upon completion of the process in thesteps S179 to S183, the process is restored to a routine at ahierarchical upper level.

As is understood from the above description, the image sensor 118exposes the imaging surface using the focal-plane electronic shuttersystem, and repeatedly outputs the images representing the object scene.The image outputted from the image sensor 118 is reduced by the zoomcircuit 124 z arranged in the pre-processing circuit 124. The memorycontrol circuit 140 extracts a part of the reduced image belonging tothe extraction area EX of a predetermined size, out of the reduced imagecreated by the zoom circuit 124 z. On the LCD monitor 148, the movingimage based on the extracted reduced image is displayed.

The CPU 136 changes the shape of the extraction area EX so that thefocal plane distortion is restrained (S127), and also, changes theposition of the extraction area EX so that the motion of the imagingsurface in a direction orthogonal to an optical axis is compensated(S133). Also, upon receipt of the zoom operation, the CPU 136 changesthe size of the reduced image created by the zoom circuit 124 z in arange exceeding a predetermined size (S179).

By changing the size of the reduced image in a range exceeding thepredetermined size, it becomes possible to execute an image-qualitycorrecting process that affects the angle of view, such as movementand/or deformation of the extraction area EX, in the entire region ofthe zoom range. This improves the reproducing performance of the objectscene image. Also, by simultaneously executing the image stabilizingprocess and the focal-plane distortion correction, it becomes possibleto shorten a time period required for the process.

It is noted that in this embodiment, the CMOS-type image sensor is used.However, instead thereof, a CCD-type image sensor may be used.

Although the present invention has been described and illustrated indetail, it is clearly understood that the same is by way of illustrationand example only and is not to be taken by way of limitation, the spiritand scope of the present invention being limited only by the terms ofthe appended claims.

1. A video camera, comprising: an imager which repeatedly outputs anelectronic image produced on an imaging surface by an exposure operationof a focal-plane electronic shutter system in a raster scanning manner;an extractor which extracts a partial image belonging to a rectangulardesignated area having a left side and a right side, out of theelectronic image outputted from said imager; an outputter which outputsa moving image based on the partial image extracted by said extractor; adetector which detects motion of said imaging surface in a directionorthogonal to an optical axis; a position changer which changes aposition of the designated area so that the motion detected by saiddetector is compensated; and an inclination amount changer which changesinclination amounts of the right side and the left side so that a focalplane distortion in a horizontal direction is restrained based on ahorizontal component of the motion detected by said detector.
 2. A videocamera comprising: an imager which repeatedly outputs an electronicimage produced on an imaging surface by an exposure operation of afocal-plane electronic shutter system in a raster scanning manner; anextractor which extracts a partial image belonging to a designated area,out of the electronic image outputted from said imager; an outputterwhich outputs a moving image based on the partial image extracted bysaid extractor; a detector which detects motion of said imaging surfacein a direction orthogonal to an optical axis; a position changer whichchanges a position of the designated area so that the motion detected bysaid detector is compensated; and a shape changer which changes a shapeof the designated area so that a focal plane distortion is restrainedbased on the motion detected by said detector, wherein the designatedarea is a rectangular area having a left side and a right side, and saidshape changer includes an inclination amount changer which changesinclination amounts of the right side and the left side based on ahorizontal component of the motion detected by said detector.
 3. A videocamera according to claim 2, wherein said detector includes an allocatorwhich allocates a plurality of blocks forming a line in a verticaldirection to the electronic image outputted from said imager, and amotion vector detector which individually detects motion vectors of aplurality of block images belonging to the plurality of blocks allocatedby said allocator, and said inclination amount changer includes afunction creator which creates a function for defining the inclinationamounts based on horizontal components of the plurality of motionvectors created by said motion vector detector.
 4. A video cameraaccording to claim 3, wherein the number of blocks allocated by saidallocator is determined based on an imaging cycle of said imager and avibration frequency of said imaging surface.
 5. A video cameracomprising: an imager which repeatedly outputs an electronic imageproduced on an imaging surface by an exposure operation of a focal-planeelectronic shutter system in a raster scanning manner; an extractorwhich extracts a partial image belonging to a designated area, out ofthe electronic image outputted from said imager; an outputter whichoutputs a moving image based on the partial image extracted by saidextractor; a detector which detects motion of said imaging surface in adirection orthogonal to an optical axis; a position changer whichchanges a position of the designated area so that the motion detected bysaid detector is compensated; and a shape changer which changes a shapeof the designated area so that a focal plane distortion is restrainedbased on the motion detected by said detector, wherein said shapechanger includes a first size changer which changes a vertical size ofthe designated area based on a vertical component of the motion detectedby said detector, and said video camera further comprises a second sizechanger which changes a vertical size of the partial image extracted bysaid extractor in association with a changing process of said first sizechanger.
 6. A video camera according to claim 5, wherein a changemagnification of said second size changer is equivalent to an inversenumber of a change magnification of said first size changer.
 7. Acomputer program embodied in a tangible medium, which is executed by aprocessor of a video camera provided with: an imager repeatedly outputsan electronic image produced on an imaging surface by an exposureoperation of a focal-plane electronic shutter system in a rasterscanning manner; an extractor which extracts a partial image belongingto a rectangular designated area having a left side and a right side,out of the electronic image outputted from said imager; an outputterwhich outputs a moving image based on the partial image extracted bysaid extractor; and a detector which detects motion of said imagingsurface in a direction orthogonal to an optical axis, said programcomprising: a position changing step of changing a position of thedesignated area so that the motion detected by said detector iscompensated; and an inclination amount changing step of changinginclination amounts of the right side and the left side so that a focalplane distortion in a horizontal direction is restrained based on ahorizontal component of the motion detected by said detector.
 8. Animaging control method executed by a video camera provided with: animager which repeatedly outputs an electronic image produced on animaging surface by an exposure operation of a focal-plane electronicshutter system in a raster scanning manner; an extractor which extractsa partial image belonging to a rectangular designated area having a leftside and a right side, out of the electronic image outputted from saidimager; an outputter which outputs a moving image based on the partialimage extracted by said extractor; and a detector which detects motionof said imaging surface in a direction orthogonal to an optical axis,said imaging control method, comprising: a position changing step ofchanging a position of the designated area so that the motion detectedby said detector is compensated; and an inclination amount changing stepof changing inclination amounts of the right side and the left side sothat a focal plane distortion in a horizontal direction is restrainedbased on a horizontal component of the motion detected by said detector.9. A computer program embodied in a tangible medium, which is executedby a processor of a video camera provided with: an imager whichrepeatedly outputs an electronic image produced on an imaging surface byan exposure operation of a focal-plane electronic shutter system in araster scanning manner; an extractor which extracts a partial imagebelonging to a designated area, out of the electronic image outputtedfrom said imager; an outputter which outputs a moving image based on thepartial image extracted by said extractor; and a detector which detectsmotion of said imaging surface in a direction orthogonal to an opticalaxis, said program comprising: a position changing step of changing aposition of the designated area so that the motion detected by saiddetector is compensated; and a shape changing step of changing a shapeof the designated area so that a focal plane distortion is restrainedbased on the motion detected by said detector, wherein the designatedarea is a rectangular area having a left side and a right side, and saidshape changing step includes an inclination amount changing step ofchanging inclination amounts of the right side and the left side basedon a horizontal component of the motion detected by said detector. 10.An imaging control method executed by a video camera provided with: animager which repeatedly outputs an electronic image produced on animaging surface by an exposure operation of a focal-plane electronicshutter system in a raster scanning manner; an extractor which extractsa partial image belonging to a designated area, out of the electronicimage outputted from said imager; an outputter which ouputs a movingimage based on the partial image extracted by said extractor; and adetector which detects motion of said imaging surface in a directionorthogonal to an optical axis, said imaging control method comprising: aposition changing step of changing a position of the designated area sothat the motion detected by said detector is compensated; and a shapechanging step of changing a shape of the designated are so that a focalplane distortion is restrained based on the motion detected by saiddetector, wherein the designated area is a rectangular area having aleft side and a right side, and said shape changing step includes aninclination amount changing step of changing inclination amounts of theright side and the left side based on a horizontal component of themotion detected by said detector.
 11. A computer program embodied in atangible medium, which is executed by a processor of a video cameraprovided with: an imager which repeatedly ouputs an electronic imageproduced on an imaging surface by an exposure operation of a focal-planeelectronic shutter system in a raster scanning manner; and extractorwhich extracts a partial image belonging to a designated area, out ofthe electronic image outputted from said imager; an outputter whichoutputs a moving image based on the partial image extracted by saidextractor; and a detector which detects motion of said imaging surfacein a direction orthogonal to an optical axis, said program comprising: aposition changing step of changing a position of the designated area sothat the motion detected by said detector is compensated; and a shapechanging step of changing a shape of the designated area so that a focalplane distortion is restrained based on the motion detected by saiddetector, wherein said shape changing step includes a first sizechanging step of changing a vertical size of the designated area basedon a vertical component of the motion detected by said detector, andsaid program further comprises a second size changing step of changing avertical size of the partial image extracted by said extractor inassociation with a changing process of said first size changing step.12. An imaging control method executed by a video camera provided with:an imager which repeatedly outputs an electronic image produced on animaging surface by an exposure operation of a focal-plane electronicshutter system in a raster scanning manner: an extractor which extractsa partial image belonging to a designated area, out of the electronicimage outputted from said imager; an outputter which outputs a movingimage based on the partial image extracted by said extractor; and adetector which detects motion of said imaging surface in a directionorthogonal to an optical axis, said imaging control method comprising: aposition changing step of changing a position of the designated area sothat the motion detected by said detector is compensated; and a shapechanging step of changing s shape of the designated area so that a focalplane distortion is restrained based on the motion detected by saiddetector, wherein said shape changing step includes a first sizechanging step of changing a vertical size of the designated area basedon a vertical component of the motion detected by said detector, andsaid imaging control method further comprises a second size changingstep of changing a vertical size of the partial image extracted by saidextractor in association with a changing process of said first sizechanging step.